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FOREWORD 


The  U.S.  Army  has  shown  increasing  interest  in  utilizing  computer  games  for 
training  purposes.  Some  games  have  been  modified  from  existing  COTS  (commercial  off- 
the-shelf)  wargames  and  some  are  being  developed  exclusively  for  various  Army  training 
activities.  An  important  issue  facing  the  military  training  community  is  that  training  time  is 
at  a  premium  and  trainees  need  to  be  able  to  play  the  game  within  as  short  a  time  as  possible 
so  they  can  start  using  the  game  to  train  the  skills  of  interest. 

The  goal  of  this  research  is  to  examine  the  usefulness  of  various  tutorial  techniques 
used  in  current  COTS  computer  games  in  helping  novice  players  reach  an  acceptable  level 
of  skill  where  they  could  play  the  game  effectively.  The  focus  of  this  research  effort  is  to 
investigate  how  two  often-used  instructional  media  (i.e.,  game  tips  and  computer-based 
tutorials)  affect  the  acquisition  of  basic  computer  game-playing  skills.  Computer-based 
tutorials  appeared  to  be  more  beneficial  for  motor  elements  such  as  maneuvering  and 
actions.  Those  who  had  access  to  game  tips  performed  better  in  cognitive  segments  of  the 
game  such  as  setting  up  game  plans  and  familiarity  with  the  game  interface.  These  results 
can  guide  the  introduction  of  computer  games  in  military  training  programs.  Future  military 
game  development  can  also  utilize  these  results  to  determine  which  type  of  instructional 
material  to  be  included  in  the  games. 

The  work  described  here  is  a  product  of  the  newly  established  post-doctoral  program 
of  the  Consortium  of  Universities  of  the  Washington  Metropolitan  Area.  The  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences,  Simulator  Systems  Research  Unit, 
supervised  this  research  effort.  The  findings  were  briefed  to  the  cooperating  agency,  the 
ADL  (Advanced  Distributed  Learning)  lab  of  the  Simulation  Technology  Center  of 
RDECOM  (Research,  Development,  and  Engineering  Command). 


KATHLEEN  A.  QUINKERT 
Acting  Technical  Director 
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UTILITY  OF  GAME  INSTRUCTIONS 


EXECUTIVE  SUMMARY 


Research  Requirement: 

The  U.S.  Army  has  made  substantia]  efforts  on  computer  games  development  for 
training  purposes  over  recent  years.  Some  games  are  modified  from  existing  commercial 
off-the-shelf  (COTS)  computer  games  and  some  are  developed  exclusively  for  various  Army 
training  activities.  As  computer  games  become  more  and  more  complex,  it  is  necessary  to 
examine  issues  such  as  how  these  games,  which  increasingly  employ  complex  plots  and 
sophisticated  response  mechanisms,  can  be  included  in  Army  training  in  an  effective  and 
efficient  fashion.  Since  training  time  is  at  a  premium,  trainees  need  to  be  able  to  progress 
from  learning  the  game  to  using  the  game  to  achieve  training  objectives  within  a  short  time. 
The  goal  of  this  research  is  to  examine  the  usefulness  of  various  techniques  used  in  current 
COTS  in  helping  novice  players  reach  an  acceptable  level  of  skill  where  they  could  play  the 
game  effectively.  The  focus  of  this  research  effort  is  to  investigate  the  effectiveness  of  two 
often-used  instructional  media,  game  playing  tips  and  computer-based  tutorials  (CBT),  for 
the  acquisition  of  basic  computer  game-playing  skills. 


Procedure: 

In  this  experiment,  novice  players  went  through  one  of  the  following  training 
regimes  for  learning  a  first-person-shooter  game:  both  types  of  instructions  (i.e.,  game  tips 
and  CBT),  only  one  type  of  instructions  (tips  or  CBT),  or  no  instructions.  Training 
effectiveness  was  evaluated  by  testing  the  participants,  after  three  hours’  of  training,  on  the 
game  interface  as  well  as  game  playing.  More  specifically,  the  training  outcomes  were 
defined  as  how  sophisticated  the  participant’s  game  plan  was,  how  familiar  the  participant 
was  with  the  game  interface,  the  game  scores,  and  whether  critical  commands  were 
incorporated  in  the  game  plan. 


Findings: 

Overall,  the  results  support  three  conclusions:  (1)  a  combination  of  both  tips  and 
CBT  are  the  most  effective  in  enhancing  overall  game  performance  compared  to  the  tips- 
only,  CBT-only,  and  the  free  play  (No  Training)  conditions;  (2)  game  playing  tips  are 
effective  in  training  novice  player  to  become  familiar  with  various  aspects  of  game  playing, 
including  game  interface  and  how  to  set  up  more  elaborate  game  plans;  (3)  CBT  can  be 
effective  in  enhancing  novice  player’s  game  scores. 
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Utilization  of  Findings: 

Army  training  that  incorporates  PC  games  will  proceed  most  rapidly  by  introducing 
the  game  through  highly  structured  computer-based  instruction  combined  with  goal  and 
strategy  oriented  tips.  The  experimental  results  reported  in  this  study  should  be  used  by 
Training  specialists  to  select  the  more  appropriate  instructional  media.  Future  military  game 
development  should  utilize  these  results  to  guide  the  development  of  instructional  material 
to  be  included  in  the  games  used  for  Army  training. 
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Introduction 


The  utility  of  commercial  off-the-shelf  (COTS)  personal-computer  (PC)  games  for 
military  training  has  been  explored  over  the  years.  One  of  the  first  efforts  was  the  Marine  Corps’ 
report  on  assessment  of  COTS  wargames  and  the  subsequent  development  of  Marine  Corps 
Doom®  (Loughran,  1999).  Although  Marine  Corps  Doom®  was  never  widely  implemented  in 
formal  training  programs,  the  Marine  Corps  has  continued  to  investigate  the  value  of  using  PC 
games  for  training  (e.g.,  Marine  Expeditionary  Unit  2000®).  The  Army  is  also  evaluating  the 
merits  of  PC  games  for  training  purposes,  which  range  from  adapting  PC  games  for  inclusion  in 
courses  (e.g.,  SPEARHEAD®  application  within  the  Armor  Captains  Career  Course  at  Ft.  Knox) 
to  investigating  first-person  shooter  games  for  cognitive  skill  practice  and  doctrinal  correctness 
(Tarr,  Morris,  &  Singer,  in  preparation).  Some  games  are  modified  from  existing  COTS 
wargames  and  some  are  developed  exclusively  for  various  Army  training  activities.  One  of  the 
examples  of  modified  COTS  game  is  FBCB2/SPEARHEADII®,  which  was  extended  from 
SPEARHEAD  II®  (co-developed  by  MAK  Technologies  and  Zombie),  an  Ml  Abrams  tank 
simulation  wargame.  MAK  Technologies  is  also  developing  a  tactical  decision-making  game, 
BC2010®,  for  training  future  Army  Brigade  and  Battalion  commanders.  One  of  the  Army’s 
most  recent  wargame  development  efforts  is  the  joint-venture  among  the  Program  Executive 
Office-  Simulation,  Training,  and  Instrumentation  (PEO  STRI),  Institute  for  Creative 
Technologies  (ICT)  of  University  of  Southern  California  and  a  team  of  game-making  companies 
(Shachtman,  2001).  Two  games,  Full  Spectrum  Command®  and  Full  Spectrum  Warrior®,  will 
be  available  in  the  near  future,  and  both  employ  urban  warfare  settings  (for  more  information, 
see  the  ICT  site  at  http://www.ict.usc.edu/disp.php?bd=proj_games).  These  games,  potentially, 
can  be  used  to  train  complex  military  leadership  tasks  ranging  from  rescuing  hostages  to  dealing 
with  social/political  issues. 

Commercially  available  PC  games  are  generally  designed  to  attract  customers;  “beating 
the  game”  is  the  main  challenge  and  reason  for  playing.  It  is  not  unusual  for  players  to  spend 
long  hours  playing  the  game  before  an  adequate  level  of  proficiency  can  be  achieved.  For 
military  training  purposes,  however,  excessive  difficulty  in  initially  learning  a  game’s  interface 
and  rules  may  degrade  or  prohibit  its  usefulness  as  a  tool  for  acquisition  and/or  skill  maintenance 
since  training  time  is  at  a  premium  in  the  military.  If  the  user  can  learn  to  play  the  game  quickly 
and  efficiently,  they  will  be  able  to  progress  from  learning  the  game  to  using  the  game  to  achieve 
training  objectives  within  a  shorter  time.  For  the  military,  the  purpose  of  using  COTS  PC  games 
is  to  both  challenge  users  and  provide  a  tool  for  acquiring  or  practicing  needed  skills.  One  basic 
issue  in  the  application  of  every  PC  game  is  ease  of  acquisition  of  game  skills  vs.  military  skills. 
In  other  words,  how  much  time  and  effort  must  be  spent  learning  the  game  interface  before  being 
able  to  exercise  the  PC  game  in  meeting  Army  training  requirements.  Therefore,  a  careful 
analysis  of  the  game  operation  training  available  in  PC  games  is  needed  if  they  are  to  be 
effective  instruments  for  military  training. 

Literature  from  video  game-related  performance  suggests  that  game  knowledge  and 
game-related  motor  skills  play  an  important  role  in  determining  game  performance  (Baba,  1993). 
Baba  found  that,  especially  early  in  learning,  game  specific  knowledge  was  more  important  than 
motor-skill  training  as  a  determinant  of  game  performance.  Novice  players  who  were  trained 
with  game-playing  tips  performed  significantly  better  than  did  those  without  the  knowledge. 
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Although  game-related  motor  skill  training  was  not  found  to  be  as  effective  as  knowledge 
training  in  Baba’s  studies,  where  a  maze-running  game  similar  to  the  popular  Pac  Man®  game 
was  employed,  it  is  questionable  if  the  same  results  would  be  found  with  different  types  of 
games.  As  computer  games  become  more  and  more  complex,  it  is  necessary  to  examine  whether 
findings  from  studies  employing  simplistic  games  are  applicable  to  those  current  games  with 
much  more  complex  plots  and  sophisticated  response  mechanisms. 

The  goal  of  this  research  is  to  examine  the  usefulness  of  various  tutorial  techniques  used 
in  current  COTS  in  helping  novice  players  reach  an  acceptable  level  of  skill  where  they  could 
play  the  game  effectively.  The  focus  of  this  research  effort  is  to  investigate  the  effectiveness  of 
two  often-used  instructional  media  (i.e.,  game  playing  tips  and  computer-based  tutorials)  for  the 
acquisition  of  basic  PC  game-playing  skills.  This  effort  will  examine  different  combinations  of 
instruction  with  a  goal  of  reducing  time  required  for  the  user  to  achieve  a  sufficient  level  of 
proficiency  to  be  able  to  play  the  game.  Literature  in  instructional  media  and  training  strategies 
has  been  reviewed  and  will  be  briefly  discussed  next. 

Computer-based  Instruction 

Literature  in  instructional  media  has  not  provided  a  consistent  conclusion  regarding  the 
superiority  of  instructional  effectiveness  of  one  certain  medium  over  other  media,  and  few  valid 
guidelines  are  available  for  instructional  designers  (Swezey  &  Llaneras,  1997).  However,  a 
substantial  number  of  studies  have  demonstrated  that  innovative  instructional  technologies  such 
as  computer-based  instruction  (CBI)  can  be  effective  in  enhancing  student  performance  and 
reducing  time  required  for  training.  For  example,  Kulik  and  Kulik  (1991)  reviewed  254 
controlled  studies  in  a  meta-analysis  on  CBI  effectiveness  and  reported  a  moderate  but 
significant  positive  effect  (i.e.,  improvement  of  .30  standard  deviations)  of  CBI  on  student 
performance.  All  studies  reviewed  involved  teaching  in  real  classrooms,  and  the  outcome 
measure  employed  most  often  was  achievement  examination  administered  at  the  conclusion  of 
the  CBI  training  program.  In  29  out  of  32  studies,  in  which  instructional  time  was  compared, 
results  showed  that  CBI  generally  saved  about  one-third  of  instructional  time  compared  with 
conventional  instructional  techniques  (Kulik  &  Kulik,  1991).  In  another  meta-analysis  of  28 
military-training  related  studies  by  Fletcher  (1990),  the  result  shows  that  interactive  video-based 
instruction  improved  student  achievement  by  about  0.50  standard  deviations  over  more 
conventional  instructional  means  (e.g.,  text,  lecture,  on-the-job  training,  videotape).  Another 
meta-analysis  of  63  studies  was  conducted  by  McNeil  and  Nelson  (1991)  on  the  effects  of 
interactive  video  on  learning,  and  an  average  of  0.53  standard  deviations  improvement  in 
performance  was  reported.  In  the  context  of  PC  game  playing,  the  above  research  findings 
suggests  that  computer-based  tutorials  may  be  more  effective  in  training  players  to  play  a  PC 
game  compared  with  other  passive  media  such  as  a  user  manual  or  paper-based  tutorial,  which  is 
employed  in  many  PC  games  developed  for  military  training. 

Part-task  vs.  Whole-task  Training 

Literature  in  training  procedural  and  psychomotor  tasks  was  also  reviewed,  since  most 
PC/video  game  playing  involves  both  knowledge  and  perceptual -motor  execution  of  game 
strategies  (Baba,  1993).  A  general  finding  from  this  literature  is  that  complex  tasks  involving 
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procedural  items  or  perceptual -motor  manipulations  greatly  benefit  from  part-task  training 
(Kirlik,  Fisk,  Walker,  &  Rothrock,  1998).  Kirlik  and  his  colleagues  suggest  that  “beginning  a 
training  program  with  part-task  training  and  then  proceeding  to  dual-task  or  whole-task  training 
may  be  the  most  efficient  training  method”  (p.  95).  In  the  context  of  PC  game  playing,  part-task 
or  segmentation  training  is  utilized  in  the  computer-based  tutorials  in  many  commercial  games. 
Those  tutorials  typically  select  a  number  of  difficulty  elements  in  the  game  and  suggest  that 
players  practice  those  elements  before  they  play  the  game.  From  the  standpoint  of  skill 
acquisition,  opportunities  to  practice  critical/difficult  elements  of  the  games  should  be  more 
effective  in  training  the  players  compared  with  simply  consulting  the  user  manual  or  playing  the 
game  without  the  tutorial.  Additionally,  games  that  do  not  have  computer-based  tutorials  cannot 
provide  their  users  opportunities  for  part-task  training  and  have  to  be  played  in  their  entirety, 
which  is  difficult  for  some  novice  players  even  with  the  game  difficulty  adjusted  to  an  easier 
level. 


A  series  of  studies,  also  known  as  the  Learning  Strategies  Project,  were  conducted  using 
Space  Fortress®,  a  video-game-like  research  tool  (Donchin,  1 995).  Some  of  these  studies 
investigated  the  differential  utility  of  part-task  vs.  whole-task  training  (Fabiani,  Buckley, 

Gratton,  Coles,  Donchin,  &  Logie,  1989;  Frederiksen  &  White,  1989;  Gopher,  Weil,  &  Siegel, 
1989;  Mane,  Adams,  &  Donchin,  1989).  A  predominant  finding  from  these  studies  is  that  part- 
task  training  is  superior  to  whole-task  training  in  terms  of  enhanced  game  performance  and 
reduced  training  time.  For  example,  Fabiani  et  al.  demonstrated  that  different  game  playing 
strategies  were  developed  depending  on  the  type  of  training  received.  The  authors  compared  the 
strategies  used  by  the  control  subjects,  who  only  practiced  the  game  in  a  whole-task  fashion, 
with  those  developed  by  the  experimental  subjects,  who  received  part-task  game-element 
training.  They  found  that  the  control  subjects’  strategies  were  qualitatively  different  from  those 
developed  by  their  experimental  counterparts,  and  the  control  subjects’  less  efficient  strategies 
resulted  in  inferior  game  performance.  In  addition,  three  of  the  Learning  Strategies  Project 
studies  (Fabiani  et  al.,  1989;  Frederiksen  &  White,  1989;  Gopher  et  al.,  1989)  report  that  low- 
ability  subjects  benefited  more  from  the  part-task  regimes  than  high-ability  subjects  in  improving 
their  game  performance.  In  fact,  low-ability  experimental  subjects  in  Frederiksen  and  White’s 
study  performed  almost  as  well  as  the  high-ability  control  subjects  playing  the  criterion  game 
following  the  completion  of  training.  The  high  ability  subjects  did  not  receive  any  part-task 
training  other  than  practicing  the  game.  Therefore,  Frederiksen  and  White  suggest  that  well- 
designed  part-task  training  may  “substantially  reduce  if  not  eliminate  differences  in  training 
performance  associated  with  starting  ability”  (pp.  110-1 11). 

Video  Game  Performance  Training 

One  study  from  the  Learning  Strategies  project  (Foss,  Fabiani,  Mane,  &  Donchin,  1989) 
investigated  the  performance  of  the  control  group  participants,  who  were  only  given  standard 
game  instructions  without  any  explicit  training  regimes.  They  found  that  although  participants 
were  able  to  continuously  improve  on  game  performance,  there  were  systematic  individual 
differences  among  participants  in  the  strategies  they  adopted.  These  different  strategies  had  a 
significant  impact  on  their  game  performance  and  the  shapes  of  their  learning  curves.  In  other 
words,  those  participants  who  figured  out  an  effective  way  of  playing  the  game  (Space 
Fortress®)  were  able  to  achieve  a  better  learning  outcome  than  those  whose  strategies  were  less 
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useful.  Another  study  from  the  same  project  (Shapiro  &  Raymond,  1989)  examined  whether 
efficient  oculomotor  movement,  which  expert  game  players  tend  to  use,  could  be  trained  to 
enhance  game  performance.  The  results  showed  that  the  group  receiving  efficient  oculomotor 
training  achieved  a  significant  higher  game  score  than  did  the  group  receiving  inefficient 
oculomotor  training  or  the  control  group,  who  simply  practiced  the  game.  Since  novice  players 
tend  to  use  inefficient  oculomotor  movement  if  not  trained  otherwise,  it  was  concluded  that 
efficient  eye  movement  training  could  benefit  novice  players  in  improving  their  game 
performance. 

Baba  (1993)  compared  the  effectiveness  of  different  instructional  techniques  for 
enhancing  PC  game  performance.  She  found  that  novice  game  players  coached  with  game 
specific  knowledge/strategies  (i.e.,  game  playing  tips  provided  by  experts)  performed  much 
better  and  had  much  steeper  learning  curves  compared  with  those  who  were  not  coached,  in 
playing  a  maze-running  video  game,  Lady  Bug®,  similar  to  the  popular  Pac  Man®  game. 
Game-related  motor  skill  training  did  not  contribute  as  much  to  game  performance.  Essentially, 
those  who  received  both  game  knowledge  and  motor  skill  training  did  not  perform  better  than 
those  who  only  received  game  knowledge  training  only.  However,  both  groups  performed 
significantly  better  than  those  who  received  motor  skill  training  only,  and  better  than  the  control 
group,  who  did  not  receive  any  training.  The  latter  groups  never  discovered  some  crucial  game¬ 
winning  strategies  on  their  own,  even  over  the  course  of  50  game  sessions.  A  related  study  from 
the  Learning  Strategies  Project  (Newell,  Carlton,  Fisher,  &  Rutter,  1989)  also  examined  different 
training  approaches  to  enhancing  video  game  (Space  Fortress®)  performance.  The  results  are 
consistent  with  Baba’s  findings  and  show  that  information  regarding  procedural  game  strategies 
is  more  beneficial  than  practice  of  the  motor  components. 

The  studies  cited  above  demonstrate  the  importance  of  game  playing  knowledge  and 
strategies  in  novice  player’s  game  skill  acquisition.  The  authors  also  provide  some  suggestions 
regarding  instructional  approaches  based  on  their  findings.  For  example,  Newell  et  al.  (1989) 
suggest  that  “optimal  approaches  to  instruction  intervention  must  combine... both  the  part  training 
of  the  response  dynamics  with  the  development  of  the  appropriate  task  strategy  for  completing 
the  task  at  hand”  (p.  214).  However,  for  games  different  from  those  employed  in  these  two 
studies,  it  still  remains  unclear  whether  the  combined  instructional  approach  (i.e.,  game 
knowledge/strategies  plus  part-task)  is  superior  to  the  situation  where  only  one  or  neither  of  the 
training  is  available. 

Military  Games 

PC  games  that  are  being  used  or  developed  for  use  at  the  Advanced  Distributed  Learning 
(ADL)  lab  of  the  Simulation  Technology  Center  of  Research,  Development,  and  Engineering 
Command  (RDECOM)  were  evaluated  based  on  the  characteristics  of  their  instructions  for  game 
playing.  The  evaluation  showed  that  PC  games  developed  for  military  use  (e.g.,  BC2010®, 
FBCB2/Spearhead  II®)  generally  lacked  computer-based  tutorials,  which  were  available  in  many 
commercial  PC  games.  Some  military-developed  games  (e.g.,  TacOps®)  used  paper-based 
tutorials  or  very  rudimentary  computer-based  tutorials,  which  did  not  provide  opportunities  for 
practice.  As  for  user  manuals,  those  provided  in  commercial  PC  games  tended  to  be  more 
detailed  while  those  of  military  games  varied  in  their  degrees  of  detail.  Finally,  none  of  the 


4 


military-developed  games  provided  game-playing  tips,  although  some  games  had  brief  paper- 
based  walkthroughs  in  their  user  manuals.  For  the  two  military-developed  games  available  at  the 
ADL  lab  (i.e.,  BC2010®  and  TacOps®),  no  game-playing  tips  were  found  on  any  military  or 
game-related  web  sites.  For  popular  commercial  games,  on  the  other  hand,  game-playing  tips 
could  be  found  on  many  game-related  web  sites  and  in  video  game  magazines. 

Based  on  the  results  of  the  above  evaluation,  a  multi-dimensional  scale  (Figure  1)  was 
developed  to  describe  PC  game  playing  instructional  approaches  and  is  depicted  below  (i.e., 
availability  of  computer-based  tutorial,  degree  of  detail  of  user  manual  and  availability  of  game 
playing  tips).  Four  games  available  at  the  ADL  lab  are  included  here  based  on  the  initial 
examination  of  their  instructional  approaches. 


Comprehensive 


Moderately 

detailed 


Limited 


None 


BC2010 


Rainbow  6- 
Rogue  Spear 


TacOps 


Spearhead 


None  Simple  Detailed  Manual  + 

manual  manual  Tips 


Figure  1.  PC  game  playing  instructional  approaches. 

Based  on  the  literature  reviewed  previously,  computer-based  tutorial  with  opportunities 
for  the  player  to  practice  critical/difficult  elements  of  the  game  (i.e.,  part-task  training)  is 
expected  to  be  superior  to  paper-based  tutorial  or  user  manual  in  its  training  effectiveness. 
Game-playing  tips  are  also  expected  to  be  beneficial  for  training  novice  players.  An  anti¬ 
terrorism  PC  game,  Rainbow  Six-Rogue  Spear®,  was  selected  as  the  experimental  tool  for  its 
availability  of  instruction  components  on  both  dimensions.  The  Secret  Service  and  the  Marine 
Corps  have  been  using  Rogue  Spear®  for  interactive  training,  and  the  Army  is  working  on 
linking  One  Semi  Automated  Force  (OneSAF)  with  Rogue  Spear®  (Bill  Pike,  personal 
communication).  The  modified  Rogue  Spear®  mainly  focuses  on  training  decision-making 
skills  at  the  small-unit  level  (Kennedy,  2002).  The  Army  is  also  evaluating  the  utility  and 
doctrinal  correctness  of  Rogue  Spear®  for  training  purposes  (Tarr,  Morris,  &  Singer,  in 
preparation). 
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Current  Study 

An  experiment  was  carried  out  to  examine  the  effects  of  PC  game  instruction  techniques 
(i.e.,  computer-based  tutorial  and  paper-based  game  playing  tips)  on  training  outcomes,  which 
was  defined  as  player’s  game  performance  after  being  exposed  to  various  instructional  methods. 
The  focus  was  on  which  combination  of  instruction  techniques  resulted  in  a  better  performance 
in  initial  game  playing. 

The  independent  variable  in  this  experiment  was  the  type  and  extent  of  instruction  for  PC 
game  playing.  Participants’  experience  in  first-person-shooter  (FPS)  game  playing  was  used  as  a 
covariate,  since  most  FPS  games  share  many  similarities  in  game  playing  and,  therefore,  the 
players’  experiences  were  expected  to  have  an  effect  on  their  learning.  There  were  three 
experimental  groups  and  a  control  group.  The  first  experimental  group,  CBT&T  (CBT  and  Tips) 
received  the  computer-based  tutorial  to  practice  essential  game  elements  and,  in  addition,  had 
access  to  the  full-length  user  manual  plus  game  playing  tips.  The  second  experimental  group, 
CBT,  received  the  computer-based  tutorial  and  had  access  to  the  full-length  user  manual  but  not 
game-playing  tips.  The  third  experimental  group,  Tips,  only  had  access  to  the  full-length  user 
manual  plus  game  playing  tips  and  played  the  game  instead  of  receiving  any  computer-based 
tutorial.  The  fourth  group,  NoTraining,  only  had  access  to  the  full-length  user  manual  and 
played  the  game  without  any  computer-based  tutorial  or  game  playing  tips.  The  four  groups  are 
depicted  graphically  below  (Figure  2). 


Full 

CBT  (CBT  + 
Manual) 

CBT&T  (CBT  + 
Manual  +  Tips) 

ComputeK 
-based  j 

1 

Tutorial  J 

None 

NoTraining 

(Manual) 

Tips  (Manual  + 
Tips) 

None 

User  Manual 

User  Manual 

only 

+  Tips 

(^User  ManuaT) 


Figure  2.  Two  dimension  training  scale  presentation  of  experimental  groups. 

The  dependent  variable  was  participants’  performance  in  the  final  test  session  after  three 
hours  of  training.  It  was  determined  that  the  performance  measures  would  include  participants’ 
game  scores,  quiz  scores  on  game  interface,  the  number  of  special  functions  utilized  in  the  game 
test  session,  and  whether  participants’  game  plans  include  a  critical  command  “Escort.”  (The 
rationale  for  selecting  these  measures  is  presented  in  the  Method  section).  Game  scores  might 
not  be  entirely  reliable  for  they  tend  to  be  contaminated  by  the  artificial  intelligence  (AI) 
employed  in  the  game.  In  other  words,  some  participants  may  have  had  higher  game  scores 
simply  because  their  plan  called  for  more  AI  in  their  games.  However,  it  was  decided  that  game 
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scores  should  be  one  of  the  measures  based  on  the  rationale  that  they  indicated  how  good  the 
participant’s  plan  was  in  addition  to  reflecting  the  action  (i.e.,  FPS)  component  of  the  game. 

Hypotheses 

Based  on  the  literature  review  and  reasoning  presented  above,  the  following  hypotheses 
were  generated, 

The  CBT&T  group  (CBT  +  manual  +  tips)  will  perform  better  than  the  Tips  (manual  +  tips) 
group,  which  will  perform  better  than  the  CBT  group  (CBT  +  manual).  The  CBT  group  will 
perform  better  than  the  NoTraining  group  (manual  only). 

These  hypotheses  are  based  on  the  research  findings  that  part-task  training  is  beneficial  for  skill 
acquisition  in  complex  tasks  and  CBTs  are  effective  in  enhancing  student  performance  and 
reducing  training  time.  In  addition,  players  who  have  access  to  game-playing  tips  are  expected 
to  perform  better  than  those  who  do  not  (Baba,  1993;  Newell  et  al.,  1989).  Baba  and  Newell  et 
al.  demonstrated  that  those  subjects  who  had  access  to  game-tips  outperformed  those  who  only 
received  game-related  motor  skill  training.  Therefore,  the  Tips  (manual  +  tips)  group  was 
expected  to  outperform  the  CBT  group  (CBT  +  manual). 

Method 

Participants 

A  total  of  44  males  recruited  from  the  University  of  Central  Florida  participated  in  the 
study.  The  ages  of  the  participants  ranged  from  1 7  to  40  (M  =  22.6,  SD  =  5.24).  Participants 
were  compensated  at  a  rate  of  $7.50  per  hour  or  class  credit  for  their  participation  in  the 
experiment. 

User’s  experience  in  PC  gaming,  specifically  FPS  gaming,  was  expected  to  covary  with 
their  game  performance  and,  therefore,  was  included  in  the  analysis.  The  procedure  for 
matching  participants  for  each  group  based  on  their  FPS  game  experiences  and  the  power 
analysis  performed  to  obtain  the  appropriate  size  of  each  group  are  reported  in  Appendix  A. 

Based  on  the  literature  reviewed  on  gender  differences  in  video  game  performance 
(Sanchez-Ku  &  Arthur,  2000;  Brown,  Hall,  Holtzer,  Brown,  &  Brown,  1997),  it  was  decided  that 
only  male  participants  would  be  included  in  the  experiment  to  prevent  additional  variance  from 
gender  differences  on  dependent  measures.  Results  from  the  studies  by  Brown  et  al.  indicated 
that  male  participants  performed  better  in  playing  a  video  game  than  did  the  female  participants 
with  a  comparable  video  game  experience  level.  Sanchez-Ku  and  Arthur  evaluated  female 
participants’  performance  in  playing  Space  Fortress®,  which  has  only  been  played  previously  by 
male  participants  in  the  series  of  studies  conducted  in  the  Learning  Strategies  project  (Donchin, 
1995).  Their  results  showed  that  their  female  participants’  performance  was  lower  than  that 
obtained  for  males  in  previous  studies,  although  both  received  the  same  training  protocol  and 
their  demographic  characteristics  and  general  attitudes  toward  video  games  were  similar. 
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Materials 


An  anti-terrorism  FPS  PC  game,  Rainbow  Six-Rogue  Spear®,  and  its  user’s  manual  and 
training  component  were  used.  The  CBTs  consisted  of  skill  training  and  course  familiarization 
training.  Skill  training  covered  drills  of  fire  and  movement,  room  clearing,  hostage  rescue,  open 
training^  and  terrorist  hunt.  Users  were  advised  to  select  a  skill  set  and  practice  it  using  various 
course  setting  (shooting  ranges,  obstacle  course,  grenade  practice,  door  breaching,  etc.).  Limited 
on-line  performance  feedback  was  available  in  the  tutorial.  Game-playing  tips  were  gathered 
from  several  Rogue  Spear®-related  web  sites.  The  tips  included  detailed  game-playing 
techniques  (e.g.,  Do’s  and  Don’ts)  and  mission  walk-throughs  with  illustrations  of  screen  shots 

from  the  game. 

The  game  consisted  of  two  phases:  Planning  and  Actions.  During  the  Planning  phase,  the 
player  created  a  plan,  in  which  elements  such  as  Roster  selection,  Kit  selection,  Team  set  up,  and 
Waypoints  planning  were  incorporated.  The  Waypoints  planning  was  the  critical  part  of  the 
Planning  phase  and  determined  the  paths  for  the  teams  not  controlled  by  the  player  (i.e.,  the 
teams  controlled  by  AI)  and  the  orders  for  those  teams.  The  game  could  be  played  in  either 
single-player  or  multiple-player  mode.  In  this  study,  only  the  single-player  mode  was  used.  A 
screen  shot  fonn  the  planning  phase  and  a  screen  shot  from  the  action  (game)  phase  are 
presented  in  Figure  3. 


Figure  3.  Screen  shots  from  Rogue  Spear®  (left:  planning  phase;  right:  action  phase). 

Procedure 

The  experiment  protocol  was  set  up  as  follows: 

Consent  and  Questionnaire  on  Computer/Gaming  Experience.  All  participants  listened  to  a 
recorded  message  about  the  experiment  procedure  and  then  filled  out  the  consent  form.  After 
consenting  to  participate  in  the  study,  the  participants  filled  out  a  questionnaire  on  their  computer 
and  gaming  background  (see  Appendix  B). 
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Initial  training  according  to  group  assignment  (2  hours).  Participants  in  the  CBT&T  and  the 
Tips  groups  were  asked  to  start  by  studying  the  game  playing  tips  (including  paper-based  walk¬ 
through).  The  CBT&T  group  then  completed  the  tutorial  and  the  Tips  group  started  playing  the 
first  mission  using  the  tips  and  user  manual.  The  CBT&T  participants  were  allowed  to  start 
playing  the  same  game  mission  with  access  to  the  tips  and  manual  after  they  finished  the  tutorial. 
The  CBT  group  started  by  going  through  the  tutorial  and  were  allowed  to  play  the  game  once  the 
tutorial  was  completed.  The  NoTraining  group  practiced  playing  the  game  during  the  entire 
initial  training  session.  The  game  included  two  components:  Planning  phase  and  Action  phase. 

In  the  Planning  phase,  the  participant  had  to  devise  a  plan  according  to  the  scenario  briefing,  and 
the  plan  included  team  member  selection,  weapon  selection,  and  waypoint  planning,  which 
included  Special  Functions  the  participants  thought  to  be  beneficial  for  the  outcome.  In  the 
Action  phase,  the  participant  essentially  executed  the  plan  he  devised  and  played  the  game  in  a 
FPS  fashion.  All  four  groups  had  access  to  the  user  manual  at  all  time  during  this  phase,  and 
those  who  achieved  satisfactory  results  on  the  first  mission  were  allowed  to  go  on  to  the 
remaining  missions.  All  games  played  in  this  segment  were  at  the  Recruit  (easy)  level. 

Practice  of  game  playing  (1  hour).  After  the  initial  training,  all  participants  continued  practicing 
the  game  at  the  Recruit  level.  All  participants  were  allowed  to  consult  their  user  manual  while 
playing,  and  the  CBT&T  and  the  Tips  groups  also  had  access  to  the  game  tips.  The  game 
statistics  were  saved  for  analysis. 

Quiz  on  game  interface  (15  minutes).  All  participants  were  administered  a  quiz  with  10  short 
questions  on  the  game  interface  (see  Appendix  C). 

Test  on  game  playing  ( about  40  minutes).  Participants  were  asked  to  complete  two  game  tests, 
which  were  the  first  mission  played  twice  at  the  Veteran  (intermediate)  level.  They  had  access  to 
the  user  manual,  but  the  CBT&T  and  the  Tips  groups  no  longer  had  access  to  the  game  tips.  All 
game  scores  (including  successful  completions  of  games)  and  time  spent  on  game  playing  were 
recorded. 

Survey  and  Debriefing  (about  10  minutes).  After  the  tests,  all  participants  filled  out  a 
questionnaire  on  the  usefulness  of  the  instructions  they  received,  ease  of  use  of  the  game 
interface,  and  other  lab-related  issues  (see  Appendix  D  for  a  sample).  During  the  debriefing 
session,  the  participants  were  informed  of  the  nature  of  the  study  and  were  also  asked  to  recall 
two  of  the  most  difficult  areas  in  their  game  learning. 

Measures 

The  following  measures  were  employed  in  the  study. 

Special  Functions.  The  number  of  special  functions  used  in  the  Waypoints  planning  was  used  as 
a  dependent  measure.  A  larger  amount  of  special  functions  indicates  a  higher  level  of 
sophistication  of  the  plan.  The  special  functions  include:  Go-codes,  Rules  of  Engagement 
(ROEs),  Speeds,  and  actions  such  as  Snipe,  Cover,  Defend,  Flashbang  usage,  Frag  grenade 
usage,  and  Breaching  charges. 
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Quiz  scores.  Participants’  quiz  scores  were  used  for  measuring  their  familiarity  with  the  game 
interface.  The  quiz  consisted  of  ten  short-answer  questions,  covering  both  the  Planning  and 
Action  phases  (see  Appendix  C).  A  full  score  would  be  10  points. 

Game  Scores.  The  number  of  kills  was  the  game  score  and  it  determined  the  outcome  of  the 
game  unless  the  participant  successfully  escorted  the  hostages  to  a  safe  zone  (i.e.,  Extraction 
Zone),  in  which  case  a  full  score  of  1 5  points  would  be  assigned.  Although  the  game  scores 
partially  depended  on  the  AI,  which  in  turn  depended  on  the  soundness  of  the  plan,  it  was 
decided  that  it  could  still  provide  a  comprehensive  indicator  of  both  planning  quality  and  game 
skill. 

Escort.  As  stated  above,  the  game  could  also  be  won  by  escorting  the  hostages  to  the  Extraction 
zone  instead  of  killing  all  the  terrorists.  Therefore,  it  was  critical  to  include  the  Escort  command 
in  the  plan  and  plot  the  waypoints  leading  to  the  Extraction  Zone  so  that  the  AI  teams  could 
execute  the  plan  accordingly.  A  plan  without  this  order  was  considered  incomplete.  More 
specifically,  a  plan  was  considered  complete  only  if  the  Escort  command  was  included  and  the 
final  waypoint  was  placed  in  the  Extraction  Zone. 

Results 

A  two-factor  (CBT  x  Tips)  multivariate  analysis  of  covariance  (MANCOVA)  was 
performed  to  analyze  the  data  with  Special  Functions,  Quiz  scores,  and  Game  Scores  as  the 
dependent  measures  and  FPS  Game  experience  as  the  covariate.  An  additional  one-way  analysis 
of  variance  (ANOVA)  and  post-hoc  tests  were  performed  to  examine  the  differences  in  the 
dependent  measures  among  the  groups.  Chi-square  tests  were  used  to  determine  if  there  was  an 
association  between  the  instruction  type  (e.g.,  Tips  or  No  Tips)  and  the  number  of  participants 
whose  plans  included  the  Escort  command.  Statistical  significance  is  reported  for  all  tests  at  a 
probability  equal  to  or  less  than  .05.  Descriptive  summary  (means  and  standard  deviations)  for 
the  first  three  measures  are  provided  in  Table  1 . 


TABLE  1.  Descriptive  Summary  (Means  and  Standard  Deviations)  for  Number  of  Special 
Functions,  Quiz  Scores,  and  Game  Scores 


Special  Functions 

Quiz 

Game  Scores 

NoTraining 

2.36  (2.20) 

4.73  (2.23) 

10.45  (2.50) 

CBT 

2.36  (3.04) 

5.32(1.94) 

11.64(1.80) 

Tips 

4.91  (1.81) 

6.82  (1.90) 

10.73  (2.90) 

CBT&T 

5.45(1.44) 

7.32  (2.03) 

12.82  (3.43) 

Note.  Standard  deviations  appear  in  parentheses. 
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The  percentages  of  participants  who  included  Escort  command  in  their  plans  are 
presented  in  Table  2. 


TABLE  2.  Percentages  of  Participants  Who  Included  Escort  Command  in  the  Plan 


NoTraining 

CBT 

Tips 

CBT&T 

Percentage  for 

18% 

.  9.1% 

100% 

82% 

employing 

Escort  command 

Levene’s  tests  of  equality  of  error  variances  were  performed  on  the  variances  of  the  three 
dependent  measures  to  examine  their  homogeneity.  The  results  did  not  show  the  variances  of 
any  of  the  measures  to  be  heterogeneous  (all  ps  >  .05).  In  addition,  Box’s  test  of  equality  of 
covariance  matrices  indicated  that  the  observed  covariance  matrices  of  the  dependent  variables 
were  not  significantly  different  across  groups.  Normality  tests  were  performed  on  all  three 
dependent  measures  for  all  four  groups.  It  was  discovered  that  for  Special  Functions,  two  of  the 
distributions  (i.e.,  NoTraining  and  CBT)  were  not  normally  distributed;  for  Game  Scores,  two  of 
the  distributions  (i.e.,  NoTraining  and  CBT&T)  were  not  normally  distributed.  Since  these  were 
only  moderate  departures  from  normality,  it  was  decided  that  ANOVAs  should  be  robust  enough 
for  testing  the  differences  (Howell,  2002).  The  kurtosis  and  skewness  scores  of  all  12 
distributions  were  within  -2  and  2  except  for  the  kurtosis  score  for  Special  Functions- 
NoTraining,  which  was  slightly  lower  than  -2. 

The  main  effect  of  “Tips”  was  significant,  F( 3,  37)  =  7.23,  p  <  .005.  None  of  the  other 
main  effect,  the  covariate,  and  interactions  among  the  factors  was  statistically  significant. 

Further  analyses  reveal  that  groups  with  Tips  employed  significantly  more  Special  Functions  in 
their  plans  than  did  the  groups  without  Tips,  F(1 ,  39)  =  19.05,;?  <  001 .  Figure  4  displays  the 
average  number  of  Special  Functions  employed  in  the  plan  for  the  four  groups.  Post  hoc  (LSD) 
tests  revealed  that  the  differences  between  the  NoTraining  (mean  =  2.36)  and  the  Tips  (mean  = 
4.91)  groups  as  well  as  between  NoTraining  and  the  CBT&T  (mean  =  5.45)  groups  were  both 
statistically  significant,  p  <  .05.  In  addition,  the  differences  between  the  CBT  (mean  —  2.36)  and 
the  Tips  groups  was  also  statistically  significant,  p  <  .05. 


Figure  4.  Average  number  of  Special  Functions  employed  in  the  plan  for  the  four  groups. 
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The  groups  with  Tips  also  scored  significantly  higher  on  their  Quiz  of  game  interface 
than  did  the  other  two  groups  without  Tips,  F(1 ,  39)  =  12.20, p  <  .005.  Figure  5  shows  the  mean 
scores  (out  of  10  points)  of  Quiz  on  the  game  interface  for  the  four  groups.  Post  hoc  (LSD)  tests 
revealed  that  the  differences  between  the  NoTraining  (mean  =  4.73)  and  the  Tips  (mean  =  6.82) 
groups  as  well  as  between  NoTraining  and  the  CBT&T  (mean  =  7.32)  groups  were  both 
statistically  significant,  p  <  .05.  In  addition,  the  difference  between  the  CBT  (mean  =  5.32)  and 
the  CBT&T  groups  was  also  statistically  significant,  p  <  .05. 


Figure  5.  Mean  scores  of  Quiz  on  the  game  interface  for  the  four  groups. 

The  groups  with  CBT  were  found  to  have  higher  Game  Scores  than  did  the  other  two 
groups  without  CBT  training,  although  the  differences  were  only  moderate,  .F(l,  38)  =  4.105,/?  = 
.055.  Figure  6  shows  the  average  Game  Scores  (highest  possible  score  being  1 5)  for  the  four 
groups.  Post  hoc  (LSD)  tests  revealed  that  only  the  difference  between  the  NoTraining  (mean  = 
10.45)  and  the  CBT&T  groups  (mean  =  12.82)  was  statistically  significant,  p  <  .05.  Figure  7 
shows  the  average  number  of  Special  Functions,  Quiz  scores,  and  Game  Scores  for  each  group. 


Figure  6.  Average  Game  Scores  for  the  four  groups. 
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Figure  7.  Average  Special  Functions,  Quiz  scores,  and  Game  Scores  for  the  four  groups. 

The  covariate,  FPS  Game  experience,  was  found  to  be  a  significant  factor  for  different 
performances  on  Quiz,  F( 1,  39)  =  5.30, p  <  .05.  Figure  8  shows  the  relationships  between 
Special  Functions  and  Tips  for  frequent  FPS  game  players  (those  who  play  FPS  games  at  least 
once  a  month)  and  infrequent  players  (those  who  never  or  rarely  play  FPS  games  and  those  who 
play  FPS  games  once  every  few  months).  Infrequent  FPS  players  who  had  access  to  game  tips 
employed  more  Special  Functions  (mean  =  4.6)  than  did  frequent  FPS  players  who  did  not  have 
access  (mean  =  2.9),  although  the  difference  was  not  statistically  significant,  p  >  .05. 

Infrequent  Frequent 

Bars  show  Means 


Figure  8.  Relationships  between  Special  Functions  and  Tips  for  frequent  FPS  game  players  and 
infrequent  players. 

Figure  9  shows  the  relationships  between  Quiz  scores  and  Tips  for  frequent  FPS  game 
players  and  infrequent  players.  Infrequent  FPS  players  who  had  access  to  game  tips  had  slightly 
higher  Quiz  scores  (mean  =  6.2)  than  did  frequent  FPS  players  who  did  not  have  access  (mean  = 
5.9),  although  the  difference  was  not  statistically  significant,  p  >  .05. 
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Figure  9.  Relationships  between  Quiz  scores  and  Tips  for  frequent  FPS  game  players  and 
infrequent  players. 


Escort  Command 

Chi-square  tests  revealed  that  the  groups  with  Tips  had  a  significantly  higher  tendency  to 
include  Escort  commands  in  their  plans,  x2  =  16.6,  p  <  .005.  91%  of  the  participants  who  had 
access  to  the  game  tips  included  critical  commands  such  as  Escort  in  their  planning,  while  only 
13.5%  of  those  who  did  not  have  access  to  game  tips  employed  the  Escort  command.  Figure  10 
shows  that  percentages  of  participants  who  included  complete  Escort  command  in  their  plans  vs. 
those  who  included  incomplete  Escort  command  for  the  Tips  groups  (i.e.,  Tips  and  CBT&T)  and 
non-Tips  groups  (NoTraining  and  CBT). 


Extraction 

^  Not  Complete 
Complete 

No  Tips  Tips 

Figure  10.  Percentages  of  participants  who  included  complete  Escort  command  vs.  those  who 
included  incomplete  Escort  command  for  the  Tips  groups  and  non-Tips  groups. 


Qualitative  Data 

Participants’  comments  as  to  the  usefulness  of  the  training  session  (i.e.,  practice,  game 
tips,  and  CBT),  the  amount  of  information  during  training,  the  ease  of  use  of  the  program,  and 
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their  own  performance  during  the  game  were  obtained  from  the  usability  survey.  A  common 
comment  from  all  four  groups  was  that  the  planning  phase  was  complicated  and  assistance  in 
addition  to  the  user’s  manual  would  be  very  helpful.  One  of  the  NoTraining  group  participant 
responded:  “I  spent  about  an  hour  trying  to  get  to  the  actual  game  playing  because  I  didn  t 
realize  that  I  was  setting  up  a  path  on  the  floor  plan  for  a  wrong  team.  I  never  played  this  game 
and  could  have  used  a  little  assistance.”  A  CBT  group  participant  responded:  The  CBT  was  a 
great  way  to  learn  how  to  play.”  Another  CBT  group  participant  stated:  “In  training  section,  the 
game  must  show  some  useful  windows  and  tips  and  how  to  command  the  teams.  There  is  not 
enough  information  available  in  the  manual....  There  must  be  a  section  where  we  can  find  the 
basic  steps  that  must  be  followed  to  achieve  certain  goals.”  Many  Tips  group  participants 
responded  that  the  game  tips  were  very  helpful,  and  one  stated:  “Walkthrough  was  short  and 
quick  and  got  me  playing  quick!”  However,  one  Tips  group  participant  responded:  There 
should  be  tutorial  missions  to  orient  the  player  to  the  game  and  controls.”  One  CBT+r  group 
participant  responded:  “The  game  tips  were  good  and  easy  to  understand.  The  time  duration  to 
get  acquainted  with  the  CBT  was  short;  perhaps  a  bit  extra  time  should  be  given.  The  program  is 
very  user  friendly.”  These  comments  suggest  that  a  combination  of  CBT  and  Tips  might  provide 
the  most  comprehensive  instructions  for  a  novice  player  to  get  familiar  with  the  game.  The 
comments  are  partially  presented  in  Appendix  E. 

Discussion 

The  goals  of  the  present  research  were  to  examine  the  effectiveness  of  PC  game 
instruction  techniques  (i.e.,  computer-based  tutorial  and  game  tips)  for  training  novice  players  to 
learn  a  FPS  game.  The  training  outcomes  were  defined  as  how  many  Special  Functions  were 
used  in  the  game  plan,  how  familiar  the  participants  were  with  the  game  interface  (Quiz  scores), 
the  Game  Scores,  and  whether  critical  commands  such  as  Escort  were  incorporated  in  the  game 
plan.  Overall,  the  results  support  three  conclusions: 

a.  a  combination  of  both  tips  and  CBT  is  the  most  effective  in  enhancing  overall  game 

performance  compared  to  the  tips-only,  CBT-only,  and  the  free  play  (No  Training) 

conditions;  . 

b.  game  playing  tips  (walkthroughs)  are  effective  in  training  novice  player  to  become 

familiar  with  various  aspects  of  game  playing,  including  game  interface  and  how  to  set 

up  more  elaborate  game  plans  by  employing  more  Special  Functions  and  game  cntical 

commands  (e.g.,  Escort);  , 

c.  computer-based  tutorial  (CBT)  can  be  effective  in  enhancing  novice  player  s  game 

scores. 

The  findings  of  this  research  suggest  that  game  tips  and  CBT  are  effective  in  quite 
different  ways.  Game  tips  seem  to  be  more  effective  than  CBT  in  enhancing  cognitive  aspects  of 
game  performances  (e.g.,  planning  and  familiarity  with  game  interface);  while  CBT,  on  the  other 
hand  is  more  effective  in  enhancing  game  scores.  Participants  who  had  access  to  the  game  tips 
employed  more  Special  Functions  in  their  plans,  indicating  that  their  plans  were  more  elaborate 
than  those  of  the  groups  that  did  not  have  access  to  game  tips.  In  fact,  infrequent  FPS  players 
who  had  access  to  game  tips  employed  more  Special  Functions  than  did  frequent  FPS  players 
who  did  not  have  access,  although  the  difference  failed  to  achieve  statistical  significance.  From 
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military  training  perspective,  however,  these  results  indicate  that  game  tips  might  be  more 
effective  in  teaching  the  trainees  how  to  develop  an  elaborate  plan  for  the  game. 

Over  ninety  percent  of  the  participants  who  had  access  to  the  game  tips  included  critical 
commands  such  as  Escort  in  their  planning,  while  only  13.5%  of  those  who  did  not  have  access 
to  game  tips  employed  the  Escort  command.  Further  examination  of  the  time  spent  reading 
user’s  manual  revealed  that  those  who  did  not  have  access  to  the  game  tips  but  employed  the 
Escort  command  spent  an  average  of  38  minutes  reading  the  user’s  manual,  while  those  who  did 
not  have  access  to  the  game  tips  and  did  not  employ  the  Escort  command  spent  an  average  of  10 
minutes  reading  the  manual.  These  results  suggest  that  the  users  were  less  likely  to  be  aware  of 
the  command  if  they  did  not  spend  more  time  reading  the  user’s  manual. 

Game  tips  were  also  found  to  be  effective  in  enhancing  Quiz  scores.  FPS  experience  was 
also  significantly  correlated  with  Quiz  scores.  The  latter  result  should  not  be  surprising, 
considering  that  most  FPS  games  share  similar  command  structures.  Experienced  FPS  players 
have  the  advantage  of  well-established  schemas  for  game  playing,  while  inexperienced  FPS 
players  tend  to  learn  the  task/interface  by  using  trial-and-error  methods  (Pillay,  2003).  What  is 
worth  noticing  is  that  inexperienced  FPS  players  who  had  access  to  game  tips  were  able  to 
achieve  Quiz  scores  that  were  at  the  same  level  as  the  experienced  FPS  players  who  did  not  have 
access  to  the  game  tips.  These  results  suggest  that  game  tips  might  be  useful  in  helping 
inexperienced  FPS  players  leam  the  game  interface  in  a  more  efficient  manner  and,  therefore,  in 
reducing  training  time. 

CBT  was  effective  in  enhancing  game  scores,  although  the  differences  between  CBT 
groups  (i.e.,  CBT  and  CBT&T)  and  non-CBT  groups  (i.e.,  NoTraining  and  Tips)  were  only 
marginally  significant.  However,  CBT&T  group  did  have  significantly  higher  game  scores  than 
did  the  NoTraining  group,  indicating  that  a  combination  of  CBT  and  game  tips  resulted  in  more 
effective  learning  of  game  playing  and  higher  game  scores.  Although  the  measure  of  game 
scores  was  contaminated  by  the  artificial  intelligence  component  of  the  game  engine,  it  still 
provides  a  comprehensive  indicator  of  game  performance,  including  both  the  planning  and  the 
action  components.  Presumably,  CBT&T  group  had  the  highest  game  scores  because  they 
benefited  from  both  the  game  tips  for  the  planning  component  and  the  part-task  tutorials  for  the 
action  component. 

Overall,  the  results  of  this  research  are  consistent  with  some  previous  studies  on  video 
game  performance  training  and  demonstrate  that  game  tips  and  CBT  can  be  effective  in 
enhancing  game  performance  (e.g.,  Baba,  1993;  Newell  et  al.,  1989).  In  addition,  our  data  show 
that  the  effectiveness  of  game  tips  and  CBT  appears  to  be  task-specific.  Tasks  involving  mainly 
cognitive  components  (e.g.,  planning  and  game  interface)  benefit  from  game  tips  while  a 
combination  of  game  tips  and  CBT  appears  to  be  most  effective  in  enhancing  the  overall  game 
performance  (i.e.,  game  scores).  As  stated  in  Newell  et  al.,  mere  game  scores  may  not  provide  a 
sufficient  basis  from  which  to  assess  all  aspects  of  game  performance.  Previous  gaming  studies 
that  only  show  advantages  of  game  tips  or  part-task  training  may  not  have  examined  all  the 
aspects  of  the  games  or  the  games  employed  in  the  studies  may  be  more  cognitive-oriented  or 
motor-oriented.  When  both  cognitive  and  motor  components  are  critical  for  the  game 
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performance,  a  combination  of  game  tips  and  part-task  training  would  probably  be  the  optimal 
approach.  This  conclusion  is  consistent  with  the  suggestion  by  Newell  et  al. 

Implications  for  Military  Training 

Due  to  the  increasing  usage  of  computer  games  for  military  training  purposes,  it  is  more 
important  than  ever  to  understand  how  computer  games  can  be  utilized  in  an  effective  and 
efficient  manner.  As  stated  previously,  one  important  issue  facing  the  military  training 
community  is  that  training  time  is  at  a  premium  and  trainees  need  to  be  able  to  play  the  game 
within  as  short  a  time  as  possible  so  they  can  start  using  the  game  to  train  the  skills  of  interest. 
The  results  reported  in  this  study  suggest  that  the  two  different  instructional  techniques  (i.e., 

CBT  and  game  tips)  seem  to  be  effective  in  different  ways,  and  players  with  access  to  both  seem 
to  learn  the  game  most  effectively.  To  be  more  specific,  CBT  may  be  more  beneficial  for  games 
involving  more  maneuvering  and  actions  such  as  those  in  FPS  games;  on  the  other  hand,  it  may 
be  more  advantageous  to  employ  game  tips  for  games  with  predominantly  cognitive  components 
such  as  setting  up  game  plans.  These  results  can  be  incorporated  in  military  training  programs 
where  computer  games  are  part  of  the  curriculum.  Future  military  game  development  can  also 
utilize  these  results  to  determine  which  type  of  instructional  material  to  be  included  in  the  games. 
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Appendix  A 

Group  matching  and  power  analysis 

User’s  experience  in  PC  gaming,  specifically  first-person-shooter  gaming,  was  expected 
to  covary  with  their  game  performance  and,  therefore,  was  included  in  the  analysis.  Based  on 
the  information  obtained  from  the  U.S.  Army  (e.g.,  The  Computer  Background  of  Soldiers  in 
Infantry  Courses:  FY  99-00  by  Fober,  Bredthauer,  &  Dyer,  ARI  Research  Report  1762),  a 
sample  that  approximated  the  range  and  level  of  experience  in  the  U.S.  Army  was  selected  and 
tested.  The  military  data  on  soldiers'  game  experiences  is  as  follows: 

Daily  1 8% 

Weekly  24% 

Monthly  14% 

<Monthly  22% 

Never  22% 

The  Daily,  Weekly,  and  Monthly  categories  was  collapsed  to  form  the  Frequent  players  category 
(56%);  the  <Monthly  and  the  Never  categories  formed  the  Infrequent  players  category  (44%). 
Potential  participants  filled  out  questionnaires  so  their  PC/video  game  and  first-person-shooter 
game  playing  experiences  could  be  classified  into  these  categories,  and  they  could  be  assigned  to 
conditions  to  replicate  the  approximate  distribution  found  in  the  Army  today.  However,  due  to 
the  extreme  difficulty  of  recruiting  college  male  students  with  limited  experience  in  PC/video 
games,  it  was  decided  that  participant’s  first-person-shooter  game  experience,  instead  of 
PC/video  game  experience,  would  be  used  to  match  participants  for  the  four  groups. 

A  power  analysis  was  performed  to  determine  the  minimum  number  of  participants 
needed  to  detect  differences  among  the  three  experimental  groups  (Shavelson,  1996).  Data  from 
Baba  (1993)  was  used  to  estimate  the  necessary  sample  size  based  on  the  rationale  that  the 
experimental  design  (i.e.,  independent  and  dependent  variables)  of  her  study  was  similar  to  that 
proposed  in  this  study.  The  four  groups  in  her  study  (Control,  Movement  training,  Game 
Strategy  training,  and  Movement  plus  Game  Strategy  training)  are  similar  to  the  groups  of  this 
current  study  (Control,  CBT,  Tips,  and  CBT  +  Tips,  respectively).  She  used  game  scores  as  the 
main  performance  measurement,  which  is  slightly  different  from  what  will  be  used  in  this  current 
study  (i.e.,  successful  completions  of  game  missions  as  well  as  game  scores)  but  is  not  expected 
to  be  greatly  dissimilar.  Therefore,  it  was  decided  that  using  Baba’s  data  should  be  adequate  for 
estimating  the  required  sample  size.  In  Baba’s  study,  there  were  4  subjects  in  each  group 
(Control,  Movement,  Strategy,  and  Movement  plus  Strategy)  and  an  estimated  effect  size  of  .94 
standard  deviation  unit  was  obtained  when  the  rate  of  improvement  was  compared  among  the 
groups.  The  improvement  scores  were  calculated  by  subtracting  the  pre-training  game  scores 
from  the  post-training  game  scores.  The  effect  size  could  only  be  estimated  because  all  groups 
were  not  given  equal  training  times.  Therefore,  the  rates  of  improvement  and  their  standard 
deviations  could  only  be  inferred  from  the  data  reported. 

The  power  analysis  revealed  that  14  participants  per  group  would  be  sufficient  to  obtain  a 
power  of  .80  (i.e.,  80%  chance  of  correctly  rejecting  the  null  hypothesis).  However,  due  to  the 
difficulty  of  obtaining  participants  with  limited  gaming  experience  (i.e.,  Infrequent  players),  it 
was  decided  that  each  group  would  include  1 1  participants.  A  total  number  of  44  participants 
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was  selected  based  on  the  percentages  in  the  five  categories  indicated  previously  but  was 
classified  as  one  of  the  two  categories  (i.e.,  Frequent,  Infrequent).  These  groupings  were 
employed  to  avoid  low  numbers  of  participants  in  the  five  categories  and  to  increase  the  power 
of  statistical  analysis.  The  44  participants,  then,  were  assigned  to  the  three  experimental  groups 
and  the  control  group  with  a  resulting  assignment  of  6  Frequent  and  5  Infrequent  participants  in 
each  group.  Participants  were  screened  to  ensure  that  they  did  not  have  prior  experience  in 
playing  Rogue  Spear®  or  Rainbow  Six®,  to  which  Rogue  Spear®  is  a  sequel. 


A-2 


Appendix  B 

Questionnaire  on  Computer/Gaming  Experience 
Participant  # _  Age _  Major - 

1 .  What  is  the  highest  level  of  education  you  have  had? 

Less  than  4  yrs  of  college _  Completed  4  yrs  of  college _  Other  _ 

2.  When  did  you  use  computers  in  your  education?  (Circle  all  that  apply) 

Grade  School  Jr.  High  High  School 

Technical  School  College  Did  Not  Use 

3.  Where  do  you  currently  use  a  computer?  (< Circle  all  that  apply) 

Home  Work  Library  Other _  Do  Not  Use 

4.  For  each  of  the  following  questions,  circle  the  response  that  best  describes  you. 

a.  Do  you  own  a  personal  computer?  Yes  No 


b.  How  often  do  you: 
Use  a  mouse? 


Daily,  Weekly,  Monthly,  Once  every  few  months,  Rarely,  Never 


ll|r® 

>  <  iJ  ’  rL  +  1  f1 


Use  programs/software  wi tii  pull-idown  menus?  '  ‘ 

^  Dail^,  Weekly,  Once  every  few^onths,  Iterely,  N^r^ 


Use  E-mail? 


Daily,  Weekly,  Monthly,  Once  every  few  months,  Rarely,  Never 

_ : - - - r' ^  r mr: ucttm; 3 rur, rq nu icz  : , r. 


Plav  computer  games?  Daily,  Weekly,  Monthly,  Once  every  few  months,  Rarely,  Never 
Play  first-person-shooter  video/computer  games? 

Daily,  Weekly,  Monthly,  Once  every  few  months.  Rarely,  Never 


5.  Which  of  the  following  best  describes  your  typing  ability?  (check  V  one) 

_ Hunt  and  peck  slowly 

_ Hunt  and  peck  quickly 

_ Type  slowly  while  not  looking  at  the  keyboard 

_ Type  quickly  while  not  looking  at  the  keyboard 


6.  Which  of  the  following  best  describes  your  expertise  with  computer?  (check  V  one) 
_ Novice 

_ Good  with  one  type  of  software  package  (such  as  word  processing  or  slides) 

_ Good  with  several  software  packages 

_ Can  program  in  one  language  and  use  several  software  packages 

_ Can  program  in  several  languages  and  use  several  software  packages 
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Appendix  C 


Quiz  on  Game  Interface 

1.1.  What  are  the  steps  for  making  Red  team’s  member  Bogart  the  leader  of  Blue  team? 


2.  What  are  the  steps  for  changing  the  primary  weapon  and  assigning  it  to  ah  team  members? 
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For  questions  4-10,  use  the  screen  shot  below: 


4.  How  do  you  select  another  weapon? 


5.  How  Hr.  von  zoom  in/out  the  map? 


6.  What  do  the  numbers  “3”  and  “2”  mean? 


7.  How  do  you  peek? 


8.  How  do  you  change  magazine? 


9.  How  do  you  change  the  Rule  of  Engagement  (ROE)  during  mission? 


10.  How  do  you  issue  the  Alpha  Go-Code  for  Green  Team? 
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Appendix  D 
Usability  Questionnaire 
Usability  Questions 

For  the  following,  circle  the  number  that  best  describes  the  degree  to  which  you  agree  with  each  statement. 


Training  Session: 


1  .  The  training  adequately  prepared  me  for  the  test  (playing  the  game  in  the  test  session). 


1  1 

1  2 

1  3 

4 

5 

6 

7 

Strongly 

Disagree 

Disagree 

Somewhat 

Disagree 

Neutral 

Somewhat 

Agree 

Agree 

Strongly  Agree 

Comment: 


2.  The  user’s  manual  was  useful  for  helping  me  leam  the  game. 

I  1  |  2  |  3  1  4  1  5  j _ 6 _ | _ 1 _ 

Strongly  Disagree  Somewhat  Neutral  Somewhat  Agree  Strongly  Agree 

Disagree  Disagree  Agree 


Comment: 


3.  The  game  tips  (walkthrough)  were  useful  for  helping  me  leam  the  game. 

I  1  1  2  I  3  |  4  |  5  | _ « _ | _ 7 _ 1 

Strongly  Disagree  Somewhat  Neutral  Somewhat  Agree  Strongly  Agree 

Disagree  Disagree  Agree 


Comment: 


4.  The  computer-based  tutorials  were  useful  for  helping  me  leam  the  game. 


I  i  1 

2 

3 

4 

5 

6 

7 

Strongly 

Disagree 

Comment: 

Disagree 

Somewhat 

Disagree 

Neutral 

Somewhat 

Agree 

Agree 

Strongly  Agree 

5.  There  was  too  much  information  in  the  training  session  to  leam  all  at  once. 


l 

2 

3 

4 

5 

6 

7 

Strongly 

Disagree 

Disagree 

Somewhat 

Disagree 

Neutral 

Somewhat 

Agree 

Agree 

Strongly  Agree 

Comment: 


D-l 


Game  Performance: 

6.  The  program  was  easy  to  use  overall. 


1  1 

2 

3 

4 

5 

6 

7 

Strongly 

Disagree 

Disagree 

Somewhat 

Disagree 

Neutral 

Somewhat 

Agree 

Agree 

Strongly  Agree 

Comment: 


7. 1  was  able  to  easily  recognize  objects  in  the  game  environment. 


l 

2 

3 

4 

5 

6 

7 

Strongly 

Disagree 

Disagree 

Somewhat 

Disagree 

Neutral 

Somewhat 

Agree 

Agree 

Strongly  Agree 

Comment: 


8. 1  was  able  to  identify  the  characters  (teammates,  terrorists,  and  hostages)  without  difficulty. 


l 

2 

3 

4 

5 

6 

7 

Strongly 

Disagree 

Somewhat 

Neutral 

Somewhat 

Agree 

Strongly  Agree 

Disagree 

Disagree 

Agree 

Comment: 

I  was  able  to  move  through  the  game  environment  without  difficulty. 

1  1 

1  2 

1  3  | 

4 

1  5 

6 

1  7  | 

Strongly 

Disagree 

Somewhat 

Neutral 

Somewhat 

Agree 

Strongly  Agree 

Disagree 

Disagree 

Agree 

Comment: 


10.  The  planning  phase  (Roster  selection,  Kit  selection,  and  Waypoint  plotting)  is  too 
complicated. 


1  1  ! 

1  2 

1  3 

4 

5 

6 

7 

Strongly 

Disagree 

Disagree 

Somewhat 

Disagree 

Neutral 

Somewhat 

Agree 

Agree 

Strongly  Agree 

Comment:  _ _ _ _____ _ _ _ 

Lab  Environment: 

1 1  I  was  able  to  concentrate  on  my  task  without  much  distraction  or  disturbance. 

I  1  I  2  I  3  I  4  |  5  |  6 _ | _ 7 _ 

Strongly  Disagree  Somewhat  Neutral  Somewhat  Agree  Strongly  Agree 

Disagree  Disagree  Agree 


Comment: 
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Appendix  E 
Participants’  Comments 


NoTraining  Group: 

1 .  It  took  a  long  time  to  get  used  to  the  game,  and  I  still  do  not  know  what  a  lot  of  it  means. 
Without  the  manual,  I  would  have  never  been  able  to  figure  out  the  game. 

2.  A  lot  of  things  to  remember.  During  gameplay,  multiple  things  are  going  on  at  once,  creating 
a  lot  of  confusing  gameplay.  The  planning  phase  takes  up  so  much  time  and  it  is  extremely 
confusing. 

3.  The  user’s  manual  had  all  the  information;  there  was  just  too  much  to  take  in  at  once. 
Knowledge  of  the  controls  and  having  practice  of  using  the  controls  is  very  different.  There 
are  a  lot  of  commands  that  are  forgotten  while  playing.  The  movement  commands  are 
uncomfortable  to  use  (my  hand  was  starting  to  cramp)  and  the  movement  of  the  sights  was 
too  fast  at  times.  The  planning  phase  would  become  easy  after  using  if  awhile,  but  it  is  very 
hard  to  pick  up  right  away.  It  was  also  a  little  hard  to  tell  the  terrorists  apart  form  certain 
objects  (lampposts,  paintings,  etc.). 

4.  I  spent  about  an  hour  trying  to  get  to  the  actual  game  playing  because  I  didn’t  realize  that  I 
was  setting  up  a  path  on  the  floor  plan  for  a  wrong  team.  I  never  played  this  game  and  could 
have  used  a  little  assistance.  The  manual  helped  me  understand  the  overall  picture  of  the 
game  but  I  mainly  figured  out  how  to  play  the  game  on  my  own.  There  could  have  been  too 
much  information  in  the  training  session  to  learn  all  at  once  for  a  novice  such  as  myself.  The 
key  functions  must  be  known  in  order  to  navigate  during  the  game.  The  planning  phase  was 
a  little  tricky  and  difficult  at  first  but  eventually  I  got  accustomed  to  it. 

.  5.  The  planning  phase  is  near  impossible!  The  section  on  planning  in  the  user’s  manual  was 
very  unclear. 

CBT  group: 

1 .  In  training  section,  the  game  must  show  some  useful  windows  and  tips  and  how  to  command 
the  teams.  There  is  not  enough  information  available  in  the  manual.  Some  important  terms 
must  be  explained,  such  as  ROE.  There  must  be  a  section  where  we  can  find  the  basic  steps 
that  must  be  followed  to  achieve  certain  goals. 

2.  The  training  did  not  adequately  prepare  me  for  setting  up  missions. 

3.  An  outline  or  table  of  contents  would  help.  The  CBT  was  a  great  example  of  game.  I  was 
unaware  of  own  team  at  first.  I  would  like  to  the  map  to  show  the  entire  location  during 
game.  During  the  planning  phase,  the  waypoint  plotting  is  too  complicated;  I  hardly 
followed  through  with  it. 

4.  The  CBT  was  a  great  way  to  learn  how  to  play.  The  planning  phase  needs  to  be  faster. 

5.  More  training  in  the  planning  segment  would  have  helped,  I  think.  The  planning  phase  needs 
more  explanation,  but  it  isn’t  too  complex. 

6.  There  was  not  enough  information  in  the  training  session.  The  planning  phase  should  be 
already  set  on  certain  levels. 

Tips  group: 

1 .  The  planning  phase  could  use  some  work  to  make  it  easier  to  use. 

2.  The  training  material  could  have  been  more  legibly  written  and  the  training  time  more 
extensive.  The  control  codes  are  not  intuitive  so  it  takes  long  to  become  intuitive  with  their 
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use.  Planning  vs.  execution — not  sure  how  correlated  (i.e.,  is  there  penalty  for  diverging 
from  plan?).  Not  being  intimately  familiar  with  firearms,  the  weapons  are  so  much  of  a  blur 
that  picking  weaponry  is  pretty  much  guesswork  (i.e.,  it  helps  but  it  takes  a  while  to  get 
working  knowledge  to  select  confidently  and  appropriately  and  one  doesn’t  know  how  if  it 
affects  underlying  algorithm,  if  at  all).  The  game  tips  (walkthrough)  were  helpful.  I  was 
frequently  referring  to  movement  codes  while  trying  to  watch  for  and  fend  off  the  bad  guys! 
I’m  not  sure  how  the  planning  phase  could  be  simplified  but  it’s  definitely  a  good  exercise  to 
have  a  plan  to  start  from.  The  up  and  down  stairs  were  hard  to  figure  out. 

3 .  There  should  be  tutorial  missions  to  orient  the  player  to  the  game  and  controls. 

4.  The  game  tips  (walkthrough)  were  very  helpful.  There  was  too  much  information  in  the 
training  session  to  learn  all  at  once;  it  was  confusing,  and  I  did  not  master  all  the  things. 
During  the  game,  it  was  easy  to  lose  orientation. 

5.  During  the  training  session,  I  learned  much  more  than  I  knew,  and  was  better  prepared  but 
not  fully.  The  walkthrough  taught  me  everything  I  needed  for  a  general  understanding.  I 
would  rather  space  out  the  time  than  do  it  all  in  one  session.  During  the  planning  phase, 
roster,  kit  and  waypoint  are  okay  but  the  alpha  plans  were  confusing. 

6.  It’d  be  nice  to  have  shortcut  key  that  showed  what  victory  conditions/successes  were.  The 
user’s  manual  was  only  useful  for  the  weapon  specifications.  Nobody  likes  reading 
instructions;  walkthrough  was  short  and  quick  and  got  me  playing  quick!  There  was  not  too 
much  information  in  the  training  session;  I  built  from  session  to  session.  Go-codes  were 
complicated  at  first. 

7.  During  the  training  session,  more  actions  and  commands  need  to  be  shown.  During  the 
planning  phase,  planning  for  actions  and  go-codes  weren’t  clear. 

8.  Best  preparation  is  familiarity  with  a  particular  mission.  I  didn’t  look  at  the  user’s  manual; 
quicksheets  covered  the  important  stuff.  The  walkthrough  was  good,  but  for  a  very  straight¬ 
forward  and  relatively  simple  mission. 

9.  During  the  training  session,  there  was  a  lot  of  info  but  not  too  much.  Set  up  the  game  was 
difficult.  The  planning  phase  have  a  lot  of  objects  and  the  go-codes  were  hard  to  understand 

and  use. 

10.  There  are  a  lot  of  controls  to  remember.  The  game  tips  were  very  helpful.  When  using  the 
program,  there  was  a  lot  to  do  all  at  once.  The  planning  phase  helps  you  to  plan  strategies. 


CBT+T  group: 

1 .  I  was  not  familiar  enough  with  key-controls.  The  user’s  manual  helps,  but  a  booklet  with 
explanations  would  be  better.  The  game  tips  were  useful,  but  only  for  the  scenario  played. 
The  CBT  help  if  I’d  known  how  to  use  them  properly.  The  planning  phase  is  not  too 
complicated— proper  planning  leads  to  higher  mission  success. 

2.  The  user’s  manual  was  very  helpful.  The  game  tips  were  good  and  easy  to  understand.  The 
time  duration  to  get  acquainted  with  the  CBT  was  short;  perhaps  a  bit  extra  time  should  be 
given.  The  program  is  very  user  friendly. 

3.  The  user’s  manual  was  okay  except  it  was  unnecessarily  full  of  not  really  necessary  info. 
The  game  tips  were  good  intro  for  the  game  and  tutorial.  Overall,  the  icon-based  things 
made  the  program  easy  but  the  planning  part  was  difficult. 

4.  The  summary  sheet  with  the  controls  was  very  useful. 

5.  I  did  not  know  what  to  expect  before  I  used  the  CBT.  There  was  too  much  to  remember 
without  playing  the  game  first. 


* 
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